Visual Estimation of Shape, Reflectance and Illumination

This work investigates one of the fundamental problems in machine vision, that of obtaining a three-dimensional (3D) digital model of a real object, from a collection of photographs. 3D computer models are a vital part of a wide range of disciplines, from the study of sculpture and architecture to archaeology, structural engineering and computer graphics. We focus on two important open questions: (a) the choice of a rich and computationally efficient mathematical representation and reconstruction algorithm and (b) coping with textureless and shiny materials. We provide answers to both questions by exploiting geometric and photometric constraints contained in the object silhouettes. We show how these features can provide sufficient information to allow the use of global optimisation as well as recover the reflectance and geometry of textureless objects. There are three main contributions in this work. Firstly, we give two novel volumetric formulations of the multi-view dense stereo problem, one based on a mesh + height field representation and the second using a binary occupancy function. Both formulations make use of a base surface which coarsely approximates the scene geometry and which is obtainable from the object silhouettes or from sparse feature matches. By using the base surface for inferring visibility and topologically constraining the scene, the approach we propose allows for surface regularisation and the incorporation of multiple wide-baseline views. The optimisation is carried out using powerful discrete optimisation algorithms such as Graph-cuts and Belief Propagation and the results are shown to be superior to traditional dense stereo methods. The second contribution is the introduction of frontier points as a powerful constraint on the scene’s reflectance and illumination. Frontier points, a geometric feature of a scene extracted from silhouettes, so far have only been used for the recovery of camera motion. In this work we show how frontier points provide a practical way of reconstructing the scene illumination and recovering the reflectance of a highly specular object. This information can then be used to obtain a 2.5D reconstruction of the object using classic photometric stereo. The third contribution is a novel technique that allows the full 3D reconstruction of textureless Lambertian objects with a small number of specular highlights. The key observation is that the object’s visual hull, the volume that maximally fills the silhouettes, provides information about the direction and intensity of a single light-source in an scene. We show how this information can be extracted via a robust voting scheme that simultaneously recovers a light source direction and locates points on the contour generator within the visual hull. After recovering the unknown directional illumination in a number of images from varying viewpoint, a novel multi-view uncalibrated photometric stereo technique is used to accurately estimate 3D shape.